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Local thresholding algorithms were first presented more 
than a decade ago and have since been applied to a variety 
of data mining tasks in peer-to-peer systems, wireless sensor 
networks, and in grid systems. One critical assumption made 
by those algorithms has always been cycle-free routing. The 
existence of even one cycle may lead all peers to the wrong 
outcome. Outside the lab, unfortunately, cycle freedom is not 
easy to achieve. 

This work is the first to lift the requirement of cycle 
freedom by presenting a local thresholding algorithm suitable 
for general network graphs. The algorithm relies on a new 
repositioning of the problem in weighted vector arithmetics, 
on a new stopping rule, whose proof does not require that 
the network be cycle free, and on new methods for balance 
correction when the stopping rule fails. 

The new stopping and update rules permit calculation of 
the very same functions that were calculable using previous 
algorithms, which do assume cycle freedom. The algorithm 
is implemented on a standard peer-to-peer simulator and is 
validated for networks of up to 80,000 peers, organized in three 
different topologies, which are representative of the topology 
of major current distributed systems: the Internet, structured 
peer-to-peer systems, and wireless sensor networks. 

I. Introduction 

Recent years have seen a surge in the number, the pervasive- 
ness, and the capabilities of networked devices, followed by 
ever greater interest in efficient algorithms for distributed com- 
putation of various functions. Some of this interest is driven 
by the performance superiority of in-network computation. 
Performance becomes increasingly important as distributed 
data becomes abundant (e.g., when processed by apps running 
on smartphones), as the balance between computation and 
communication costs tilts in favor of the former (e.g., in a peer- 
to-peer environment), and when energy conservation becomes 
a major concern (as in a wireless sensor network). Additional 
causes for this interest are more social than technological: 
A distributed architecture - harder to manipulate and control 
by any single entity - is preferred where users distrust any 
third party, and where privacy is a concern. This is frequently 
the case where there is suspicion of bias (e.g., in shopping 
recommendations) or in semi-legitimate applications (e.g., file 
sharing). 

In-network computation algorithms fall into several cate- 
gories. Those categories which provide proof of correctness 
include broadcast and convergecast based algorithms such 
as those implemented using MapReduce 1271 or tailor-made 



methods im, gossip algorithms fT6\, (5], fU], fT\, S, 
ifTSll . M, ES, 113, and local thresholding algorithms EH, 
EH, ll30l . iHl, O. Broadcast and convergecast require global 
coordination, which can be costly. When dealing with dynam- 
ically changing data and topologies, as in peer-to-peer and 
wireless sensor networks, such algorithms are not attractive. 
Gossip algorithms require no coordination and are suitable 
for dynamic data and topologies. However, the correctness of 
gossip algorithms relies on rapid mixing of the inputs, which 
is, by definition, communication intensive. 

Local thresholding algorithms are a third category of in- 
network computation algorithms. Unlike gossip algorithms, 
they focus on decision rather than approximation problems. 
For instance, a basic local thresholding algorithm [31] would 
compute a majority vote where its counterpart gossip based 
algorithm computes the average. Research has shown that 
many data mining algorithms (e.g., associations, decision trees, 
approximated clustering) can be mapped to large numbers of 
convex thresholding decisions. 

At the heart of any local thresholding algorithm lies a local 
stopping rule: A condition computed by each peer on its data 
and the messages it received and has sent. When the condition 
is violated, the peer must send out messages in prescribed 
ways. However, when the condition holds, the peer does not 
have to send any further messages because either every peer 
currently computes the correct outcome, or there is a peer in 
the network whose condition is violated and who is responsible 
for correcting the computation outcome of all peers. Since 
local thresholding algorithms rely on achieving local balance 
rather than on mixing the inputs they are the most commu- 
nication thrifty of the three categories. However, all local 
thresholding algorithms presented until today require cycle- 
free routing, which makes them very difficult to implement in 
real distributed systems. 

This work is the first to present a local thresholding al- 
gorithm suitable for general network graphs. The algorithm 
relies on a new local stopping rule and on new update rules 
that restore local balance when the stopping rule fails. Unlike 
those used in previous local thresholding algorithms, the new 
rules do not rely on cycle freedom for their correctness. The 
new stopping and update rules are general and can replace 
any of those used in existing local data mining algorithms. 
Additionally, because they can handle cycles, algorithms based 
on the new rules can handle partial message failure. Of no 
less importance is the representation of the problem in a new 
mathematical framework, which simplifies proofs and invites 
further development of local thresholding algorithms. 

The rest of this paper is organized as follows: First, Sec- 
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tion in] describes the notation used and the formal problem 
definition, as well as the success metrics. Section HIH provides 
the main mathematical contribution - the new stopping rule. 
Section |IV] complements Section |lll] by describing a general 
and accurate balance correction method. Section |V] combines 
those two contributions to form a useful algorithm. Thorough 
experimentation is described in Section [VT] Next, Section IVllI 
explains the relation of previous work to this one. We conclude 
with some open reseaixh questions in Section IVIIII 



II. Notations and problem statement 

A. Mathematical notation 

Much of the math in this paper uses weighted averages. To 
simplify notation throughout the paper we adapt the notation 
proposed in |12 | in which © denotes the weighted average of 
a pair and the weighted average of a set of vectors. The 
scalar multiplication of a weighted vector, which affects its 
weight, is denoted 0. Formal definition follows. 

Definition 1. [Weighted Vector Space] Let V be a vector 
space and C a corresponding field of scalars. Denote + and ■ 
the addition and the scalar multiplication of V. The weighted 
vector space W with addition © and scalar multiplication 
is defined as follows: 

• The elements of W are pairs (T^,c) such that 17" G V 
and c G C 

• The scalar field of W is C 

• The scalar multiplication is defined 



Cl 



(17',C2) = (l^,Ci ■ C2) 



The addition is defined 



(^,Cl)©K,C2} = 



Cl 



Cl + C2 



Wl + 



C2 



Cl + C2 



■V2,Ci+ C2 



To be precise, j^^r^ denotes the multiplication of ci in the 
scalar inverse of ci +C2. The obvious example for a weighted 
vector space is one in which V is R'' and C is M. However, 
the definition of the weighted vector space is general enough 
to include many types of V and C. Of special interest is that 
for a specific space of random vectors V the corresponding 
C can be the space of covariance matrices. This has many 
applications in data mining and machine learning, including 
z-score normalization. 

It is easy to validate that the weighted vector space W with 
the operations and is a vector space and that any Xq 
whose weight is the zero element of C is an identity element 
of this space. Also, the triangle inequality with respect to the 
L2 norm ||-|| holds for the vector part of the weighted vector 

For brevity, we make four additional notations: We will refer 
to the vector part of any X G W as ^ and to its scalar part as 
\X\. Additionally, the additive inverse operator is denoted by 
where X y = Z if and only if X = F Z. Finally, for 
a set S" = {Xi,X2, . . . , X„ : G W} the additive iteration 
^1 © ^2 ® ■ • ■ © -'^n is shorthanded to ©jf .gg X^. 



B. Problem statement 

Let P = {pi, . . . ,pn} be a set of peers with inputs X = 
{Xi, . . . ,Xn}, respectively. Let A'^ C P be the set of peers 
connected to pi. The average input is ®p gp Xi or just X 
for short. The objective of the peers is to all compute a function 



./ [® ■ Peers communicate by sending messages, where 
each message consists of a single weighted vector The latest 
message sent by pi to a neighbor pj G Ni is denoted Xij. 

Throughout this paper it is assumed that both the set of 
peers, P, their inputs, X, and the connectivity of each peer, 
Ni, vary over time. Link failures are modeled as changes 
in the neighbor sets of the peers at both ends of the link 
and peer failure as failure of all its links. The method of 
failure detection is not specified and it is sufficient that 
failures are eventually detected (i.e., a heartbeat mechanism 
is sufficient.) This paper assumes symmetric communication, 
i.e., that pj ^ Ni ^ pi G Nj. Our proofs of correctness also 
assume communication is ordered and reliable. However, we 
show (in Section [V| how sufficient ordering can be enforced. 
We further show experimentally that limited random message 
dropping does not affect correctness in any serious way. 

Let /; be a function which determines the output of peer 
Pi. An algorithm provides eventual correctness if, whenever 
changes cease for a long enough period, it guarantees all fi 
will converge to / ^0 X^ as computed on the current set 
of inputs. Often, however, changes never cease for a long 
enough period, and intermittent accuracy - the percent of peers 
which compute the correct outcome - is more important than 
convergence. An algorithm is local if the resources every peer 
requires in order to arrive at a prescribed level of intermediate 
accuracy tend to a constant when the number of peers tends 
to infinity. 

Under the said assumptions we denote the agreement of 
Pi and its neighbor pj G iV^ by Ai_j — Xij ffi Xj^i. 
We note that unless messages are still traveling between 
Pi and Pj, Aij — Aj,i. The state of pi is denoted 

Si ^ X, (S (0p^g^J^j,i e Arij)^ The function each 
peer computes, fi, is defined throughout the paper to be 
fi — f (^Si^. Finally, the function / considered in this 

paper is / : K'' 7^ U {nil}, where 7^ = {i?i, i?2, • ■ • } 
is a (possibly infinite) set of convex regions in K** and 
nil = M.'^ \ Ri their complementary region, and where 

/ (^) = l^^'S™ e 7^ : ^ G P} if^ e[j^^R^ 



nil otherwise 
A specific case of interest is where 7^ is a convex cover of 



III. A LOCAL STOPPING RULE FOR GENERAL NETWORK 
GRAPHS 

The correctness of the new algorithm is based on a new 
stopping rule whose proof does not require that the network 
be cycle free. We first show that throughout the workings of 
the algorithm, the average of the inputs is reflected in the states 
the different peers maintain. This provides that the global input 
is preserved in the states regardless of how it is distributed by 
the algorithm. 
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S^ = 



Theorem 2. [Mass Conservation ( /[Wl Proposition 2.2)] The 
average of the states of all peers is equal to the average of 
the inputs of all peers; i.e., 0p.gp Si — 0p.gp Xi. 

Proof: The average of all states is 0^, 
ing the right-hand side 

Since every Xi,j appears inside the square brackets 
twice, once preceded by © and once by 0, we have that 



yields 



0r 



Rearrang- 







P,eP 



e 



^P^eP 



0. In other words, it is an identity element. It follows 



that 



PieP 



Si — 0piep X 





Next we prove our main theorem, which is a local stop- 
ping rule that makes no assumptions on the network graph 
properties. Local stopping rules are the cornerstone of local 
thresholding algorithms. The importance of the local stopping 
rule is that each peer can verify its conditions and then stop 
communicating, relying on the correctness of the stopping rule 
to provide that one of two things will happen: Either all peers 
stop communicating, and hence the local status represents the 
global average, or there must be a peer which does not stop 
communicating, and hence computation will continue. 

Theorem 3. [Local Stopping Rule] If for all pi e P, every 
Pj e Ni, and fo r some co nvex R C W^, it hol ds th at 



\Si Q Ai 



^ SiO Aij e R and \ Aij\ ^ ^ Aij e 



R, then ^ X e R as well. 

Proof: Let T {P, E) be a spanning tree over the network 
graph rooted at an arbitrary peer Consider a convergecast 
process in which, starting in the leaves, every peer pi waits 
until it receives a message from all of its descendants and then 
sends a message to its parent pj which increases the value of 
the agreement Ai^j by Si. 

First consider a leaf pi. The leaf waits for no incoming 
messages and sends a message to its parent pj. To set the new 



message. Therefore, by induction, the same happens when a 
non-leaf peer pi receives the final message from any of its 
descendants and sends the message to its own parent pj. 

Finally, when the root of the spanning tree, call it po. has 
received all of the messages, we have Si — for any pi ^ po 
and 5o £ R. From Theorem |2] 0X = 0p gp Si and since 
in 0p^ 5, = 5*0 ® 5'i © ■ • • © the latter n- 1 elements are 
zero elements, we have that @X — 0p.gp Si = Sq. Thus, 

^ X — So & R, which proves the theorem. ■ 
We stress that the stopping criterion does not require that 
the graph be a tree. It is important that the reader not be 
confused by the spanning tree in the proof of the Theorem. 
Using a spanning tree is a methodology, not a condition for 
correctness. 

IV. Balance correction 

A local stopping rule is just one part of a local thresholding 
algorithm. It must be complemented by a method for achieving 
the conditions set by the local stopping rule. In this section we 
first prove that for any set of inputs, any network topology, and 
any convex region R containing X, there exists a solution 
which satisfies the conditions set by Theorem [3] Then, we 
describe a pair of local correction policies, each of which is 
a formula by which a peer can compute outgoing messages 
such that after the messages are sent the conditions hold at 
that peer 

A. Existence of a solution 

We first show that regardless of the topology and the input 
of the different peers, there is a set of values for the different 
Xi,j such that all non-zero Si Q Aij are equal to all non- 
zero Ai,j. This means that they all must reside in the same 
R, regardless of its shape. Consider any spanning tree of the 
network graph. The solution is to assign zero weight to Xij 
and Xj,i corresponding to edges not on a spanning tree. The 
other Xij and Xj,i will be assigned values which provide that 



A'^. to S^, the content of the message should be 5,©X,j. forevery^ ePandpj e iV, the following holds: S'j© A, j = 



This way A[ .j = Si © Xi^j © Xj^^ = Si (B Aij. Sending the 
message increases Xij by Si and this increase is deduced from 
the current Si, which results in a new status S'^ — SiO Si = 0. 

Now, consider the change as it is experienced by pj. Its 
agreement with pi changes from Aij to Aij © Si. However, 
following from the triangle inequality^and the convexity of R 
(see Section III- Ab . it follows from Aij and 5, being in R that 
the new agreement A'^ ^ — Si ® Aij is in R as well. 

The status of pj, Sj, also changes as a result of the change 
in Xij. Since X- j 



Q®X. 



Xij © Si, the new value of Sj, S'^, is 



Sj = Sj © Si. Again, since Si and Sj ai'e in R, so is 5*^ = 
Si © Sj. Finally, for any other neighbor of pj, Pk ^ Pi ^ Nj, 
the value of Sj OAj^k is increased by Si. Since Sj © Aj,k G R 
and ^i e R, S'^ © Aj^k = S, © Sj © Ajl e R. 

We conclude that when a leaf pi sends a message to its 
parent pj Si becomes a zero element of W and the conditions 
of the theorem continue to hold for pi and pj. The same 
happens when all of the leaves below any peer send their 



Theorem 4. [Termination state existence ] For any connected 
netwo rk graph and any set of inputs and a convex R such that 
X G i?, there is a setup of values for Xij such that the 
conditions of Theorem |5] are met. 

Proof Let T {V, E') be a spanning tree over G (V, E) and 
let Vi be the vertices in p^'s subtree. Let the global weighted 
average be 0X. For every {pi,Pk) G E such that {pi,Pk) ^ 
E', let \Xi,k\ = \Xk.i\ ~ 0, which satisfies the conditions of 
Theorem [3] 

Define the subtree status Yi recursively as follows: For 
a leaf node, the subtree status is equal to the input, Yi = 
Xi. For a non-leaf node, Y^ is the status omitting the last 
message sent to and received from its parent, Y^ ~ Xi ® 

Now, we define a message which should be sent and one 
which should be received from each node to any neighbor 
on the spanning tree. Let pi be a node and pj its parent on 
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the tree. We set X^j = 5 li © -'^ and X^^i 



If Pi is a leaf, then Xij ^ ^ QY^Q ^ 

4M 



©ex = i0 

G^XO^QXi. It follows 



XiQ^Q^X andXj.i 
that Si © A.J- = J = X^J © Xjv< = 2M and that 
5,; = 2 yli J = © ^. Thus, the vector parts of S^, 

of Aij and of Si are all equal to 0X. Additionally, 

XiJ Xj- , = X, |iy X. 

Next, consider a peer connected to 

\Ni\ — 1 leaves and to a parent pj. Thus, pi 

has = X, © (0p,^p^eArjXfc,^0X,,fc)) - 

^<©(0p,5:tp,eiv, (^fe 9 M © 0^))- Hence, computes 
with its own parent pj Xi j Xj i — Yi Q Q ^ X = 

ey 9 7^ © -''^^ which is the difference between the 
average of the inputs in p^'s subtree and the global average. 
Applying induction, we get that for any pi and its parent pj 



X^ (0p,^p,eAr, {Xk,. ^ © 

®P,^P.ev,{Xkej^Qex)) and 



,Gy, Xk I |y 



©^00^- 



we have Yi = 

Xi 

Xi,3 Xj^i = {®pk 

It follows that for every peer pi and any pj (parent or non 
parent), we have A^.j = SiQ Aij = X, and thus 

Aij = Si Aij — 0X, which agrees with the 
conditions of Theorem |3] regardless of what R is. ■ 

We note, once more, that although the proof of Theorem |4] 
uses a spanning tree, the correctness of the stopping rule does 
not rely on the topology. The proof only serves to show that 
at least one global termination state exists. 

However, a peer cannot directly compute this specific ter- 
mination state unless it has knowledge of the global topology 
and of all inputs. Therefore, we now move to describe how 
local correction methods can be directly computed by a peer 
using only its own state and agreements with neighbors. These 
methods do not necessarily provide global termination. Rather, 
they restore the stopping condition at the peer which uses 
them, while possibly by violating the condition of its neighbor 



B. Local correction 

Consider a peer pi whose state currently violates the 
conditions of Theorem [3] i.e., for some of the pj G Ni 
either Ai^ ^ R and \Aij\ > or Si G Ai^j ^ R and 
\Si Aij\ > 0. A local correction heuristic is a method for 
computing messages for some of the peers in Ni such that the 
new values of the respective Xij's cause the conditions of 
Theorem [3] to hold for pi. Since this paper describes a general 
algorithm, we are interested in local correction methods whose 
success does not depend on the state of the peer, the topology, 
and the specific R in question. 

A simple way to achieve independence of R is to make sure 
that after the messages are sent, all Aij and all Si Aij 
are equal. This way, if one is inside R, then all are inside 
R and tl^ e c ondi tions of T^heo rem [3] are ^ met . First, p ote th^ t 



S, 



1 Aij — Ai_j means 5*,; 



A. 



i.e., the local correction method must compute new values for 



the Xi,j of Pj G Ni such that 

Vpj e iV, : ITJ = st. 



(1) 



Theorem 5. [Perfect correction ] Let A[ ^ and S[ denote 
the new values computed by changing each Xij to X[ 

Then A[ j — S'i for all pj G Ni if and only if A[ j = 

© (X, © 0p,ejv, 2 © Xk, 



Proof Because the different A'^ j are all equal to j, we 
write for all Pj,pk G Ni: 



A' 



A' 



XjA — X-j. Xk^i 



X' 



(X,' , X,- ,) Xk, 



Xk,rexl^ = {2QXk^,)e{xl^®x,^,) 



— S; 



The equation Vpj G Ni : A[ j = S'i can be rewritten 



Vpj G N, : X:^- X,- , X, {Xk,^ XI,,) 

PkeN, 

> 

= x^0 [{2 Q Xka) e {xl^ ® X,,:)] 

PkeNi 

> 

= X,0 [2QXka]e (X(,ffiX,,,) 
PkeN, PkeN, 

> 

= X,(B [2 Xk,^] © [|iV,| (X^ X,- ,)] , 

Pk^Ni 

which is equivalent to stating, Vpj G Ni, that: 

> > 

^ X© [2QXk,^] 

PkeNi 



(|A^,| + 1)0(X^0X,-,) 



Since multiplication by a constant only changes the weight, 
not the vector part of a weighted vector,we have that 



Vp, G N, : Xl^ X,- , = X, [2 Xk^,] (2) 

Pk&Ni 

The set of possible A'^ j which satisfy the requirement in 
Eq. [T]can be computed by normalizing the weighted vector to 
the desired [A' J : 



A' 



I A' I 



^^©0p,eAr,2 0X, 



Q\X,® 20 Xka 



Pk&Ni 



(3) 



In other words, what Theorem |5] states is that if the peer 
chooses a new weight for its agreement with a neighbor, \A^ ^ |, 

then that weight dictates the value of j which would satisfy 

Eq. [1] Since the peer can enforce both \A^ j\ and A'^ j by 

choosing appropriate \X^ j\ and X'^j, Theorem |5] identifies a 
range of outgoing messages that satisfy this Eq. [T] which are 
those which can be computed using Eq. [3] 
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C. Weight distribution schemes 

Theorem |5] provides the general solution to the problem 
of computing outgoing messages that will set p/s state to one 
which obeys the conditions of the stopping criterion, described 
in Theorem |3] However, the equation does not dictate a single 
solution but rather a proportion between the weight o f th e 
outgoing message, | X'^ j | , and the vector of that message, X'^ j . 
Each choice of I X' , I will have a different effect on the states 
of the sender and the recipient. 

The most important property to be had in a correction 
scheme is guaranteed convergence to a global solution. From 
results obtained for other iterative averaging algorithms |14|, 
we know that it is simple to achieve convergence of the \Si\ 
of all Pi to a constant value that depends only on the topology 
: each time a peer's state violates the stopping criterion, it will 
distribute half of \Si\ to its neighbors. So the total weight to 
be distributed is given by the equation 



E 



{\X'J - \X,^,\) - 



\S^\ 

2 



(4) 



However, naive implementation of this poUcy may lead to very 
small \Si\ values at some peers. While mathematically small 
weights do not pose a problem, in practice they would lead to 
numeric instability. Therefore, a minimal weight f3 is enforced 
on 1^,1 by using ~ I^^^^D = We find 

that a small (3 does not hinder convergence. 

1) Uniform weight distribution: A simple weight distribu- 
tion policy is to allocate a constant portion of to each 
Pj e N„ i.e., to set \X'^ ^\ - |X,,,| = ^fj^ . Since , ' 



\X,, 



the new L4' 



1,3 I 

\A, 



— "'^ l^HJ] — 2\Ni\ ^ \^'-i-J\ 

We denote this the uniform weight distribution method and 
formally define it by instantiating Eq. |3] to: 



A' — 



2lNi\ 



X, 



2 Xk, 



X, 



) 2 0Xfc,,; 



(5) 

2) Selective local correction: By distributing the weight 
uniformly, as described in Eq. |5] above, a new value is 
computed for every Xi,j. This is often unnecessary as many 
of the neighbors pj G Ni may have Ai,j and Si Q Ai,j that 
still fall inside R. Setting the Xij of those neighbors to a 
new value can be doubly wasteful: a message must be sent 
to every neighbor pj, and the change of Xij might well 
change some Sj Aj,k at pj to the degree that it violates 
the stopping condition, triggering further messages. A solution 
which selectively sends messages to only part of the neighbor 
set Ni and still brings all of the Aij and Si Ai,j into R is 
therefore desirable. 

Denote Vi the set of neighbors for whom th e stopping condi- 
tion is violated, Vi = |pj G N.^ : Aij ^ RV SiQ Aij ^ 
The complementary set, Ni \ Vi, are the neighbors whose 
update may be avoided. Consider an imaginary peer pim 

— Xj^i for all 



with N, 



V,, X,,, 



Pj G N.i, and Xim = X^ 



X,^j and X 



that Xi, 



'0 



2QX, 



Sii 



A, 



Note 



Si 0p^.gy, Ai^j. If Pim sets every Ximj according to Eq. 
12] then we have 



A' 



(ySira ®p^izMi„^ -^i 
■ (^Si 0p^gy^ A 



_ \A'ij\&{Xi®i 



(Xfc,,eXi,fc)g 



\Xi(: 



(6) 
(7) 

(8) 



It then would follow that if for all pj G Nm (i.e., in V), 
^im j (which is equal to A'^ j) is set according to the policy 
detailed in Eq. |5] then \S'iJ = \Si\ = The selective 

version of the uniform weight distribution policy is therefore: 



A' 



I A 



S^ 



Pk€V^ 



(10) 



The problem with a selective policy lies with the neighbors 
that did not violate the stopping condition, those in N \Vi. 
Since setting the Xi^k of each pk G Vi changes the status Si, 
it may well be that for pj G Ni\Vi the vector Si Aij is no 
longer in R. One solution can be to iteratively add neighbors 
to Vi if they violate the stopping condition, and to terminate 
the iteration when V no longer grows. At this stage only, 
messages are sent to all of the neighbors in Vi. Note that, at 
worst, iteration ends with Vi = N, which is the non-selective 
solution. 



V. Source selection 

The stopping and correction methods presented in the 
previous sections are general. They can be applied to various 
sets of convex regions. In this section we demonstrate their 
application to the problem of source selection. The source 
selection problem is a generalization of the majority voting 
problem in which votes and options are vectors in R'^ rather 
than the points {0, 1}. The generalization is sufficiently rich to 
allow reduction from data mining problems such as decision 
tree induction f4^ and fc-median fTE\, and yet simple enough to 
allow thorough and application-independent experimentation. 

Let C — {ci, C2, ■ ■ ■ ,Ck G M"*} be a set of options and let 
Xi be the input of pi such that \Xi\ = 1 and G M''. 
The objective of the peers in the source selec tion ^ problem is 
to compute / ^0X^ ~ argmiuc-gc | Q ^ 0-^^11^ where 
the norm || • || can be any norm (we use the L2 norm). Note that 
n = {[ci] , [C2] , . . . , [ck]}, where [c,] = {x G M : / (x) = cj 
is a set of convex regions, and nil, the complementary region, 
is emptjQ. Thus, we can apply local thresholding to the 
problem with the new stopping rule and the local correction 
policies described in Section IIV-BI 

To solve the source selection problem, peers follow the gen- 
eral setup presented above. They exchange weighted vectors 
and retain the latest vector sent and the latest received from 

'To see that this problem reduces to a majority vote, consider Xi G {0, 1} 
and C = {0, 1}. 
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Algorithm 1 Local Source Selection in General Network 
Graphs 

Common inputs for all peers: /3 G [0, 1], ^ e N, C = 

{co,ci,...,Cfc e R'^} 

Private input of p,: e W^, N,cV 

Output of k: / 

Initialization: 

X, ^ (St, 1), it < seqi ^ 

For all pj e Ni 

- ^ (0,0) 

- /asij ^ 

On a message {X, seq) from e Nil 

If seq > /astj then 

- lastj seq 

- Xj^i <— X 

- Si -(r- Xi (B ®p^. gjVi (-^j-i © ^ij) 

On initialization, on any change to Si, and on timer 
expiration: 

If currentTimei) — £i < £ 

- Set timer to currentTimeQ + £ ~ £i and return 

^ [p, £N,:f (a^) ^ / (st) V / (a,,, e A, j) 7^ / (st)} 
If = return 

Do 

- newSi ^ oldSi © 0p^.gy^ 

- \/pj e do J- ^ " i^e^g^ newS'i 1 Xj,i 

' Si ■'r- Xi ®p^. g^r. (^j.i ^ij ) 

- Wi ^ [p, G Af, : / 5^ / (5t) V / (A,,,eA,j) 7^ / (st)} 

- ^ U W', 
While VF^ 7^ 
seqi ^ seqi + 1 

£i <r- currentTimei) 

For all Pj G send {Xi j, seqi) to 



each neighbor. They maintain and update Aij and Si with 
every incoming message and every change in their input, and 
they react to violations of the stopping rule by computing 
corrective messages. 

To this general framework, the local source selection al- 
gorithm for general network graphs (LSS, for short) (Alg. 
[T]) makes three modifications. First, it always evaluates the 
stopping rule with respect to the convex region f (^Si^. 
Second, it attaches a sequential number to every outgoing 
message, so that the recipient can ignore late arrivals. This 
is because, in a real system and in our simulations, messages 
do not necessarily arrive in the order in which they were sent. 
Finally, it employs a leaky bucket mechanism which ensures 
that messages are not sent at a rate higher than once every £ 
time units. 

VI. Experimental validation 

The main contribution of this work is not a new algorithm 
for a specific data mining problem but rather a fundamental 



change which can be applied to all existing local thresholding 
algorithms. The purpose of the following section is to evaluate 
the effect of the main arguments of the problem: The system 
and its dynamics, the input domain V and the output domain 
TZ, and the algorithm parameters, on performance. 

To make evaluation more specific, it is carried out in 
the context of the LSS algorithm. However, performance on 
specific applications and in specific system settings would have 
to be evaluated per case. To facilitate that, the experiments 
were carried out with a standard simulator, peersim [E4l . and 
the code is available on-line. 

A. Experimental setup 

LSS performance is influenced by four types of parameters. 
The first are those of the system: the number of peers, n, the 
topology in which they are arranged, and the message drop 
rate, r. In our experiments we use three different topologies: 
To investigate unstructured peer-to-peer systems we use the 
Barabasi-Albert [1] model, which is a well-known approxi- 
mation of the Internet router topology and which is claimed 
to approximate the structure of systems like Gnutella ||281 . To 
investigate structured peer-to-peer systems, we use the popular 
Chord topology [291 with the variant that connection with 
fingers is assumed bidirectional (in essence, implementing 
Symmetric Chord ['231). For the third target topology, wireless 
sensor networks, no standard accepted model exists, and mere 
connectedness is a design challenge |20|. We therefore opt for 
a wireless sensor network in which sensors are locations on a 
bi-dimensional grid. In all those three topologies, the average 
connectivity \Ni \ can be controlled, although in Chord its value 
is typically much larger (log (n)) than in the other two. We 
experiment with reliable communication and with a range of 
message drop rates. 

The second type of parameters are those related to the 
function computed, which in the case of LSS are the number 
of different sources from which the selection is made, k, the 
dimensionality of the data, d. and the distribution of the data. 
In our experiments, the data is normally and independently 
distributed along each dimension. We randomly select one of 
the possible sources as the desired outcome of the algorithm 
and denote the contender its nearest neighbor The mean of the 
data is set to a weighted average of the desired outcome and 
the contender. The weight given to the contender, between 
zero and one-half, is denoted the bias of the data. The 
standard deviation of the data is selected as a multiplier of the 
distance between the desired outcome and the contender That 
multiplier is denoted std. When the data is dynamic, inputs 
are resampled from the same distribution at every cycle. The 
proportion of peers whose data changes at every simulation 
cycle is denoted the noise rate, which is measured in units of 
changed peers per million simulation cycles (ppmc). 

Figure [T] depicts an example of two hundred data points 
with d = 2, k = 3. The mean of the data is denoted by the X, 
which is located at a bias of 40% between the desired outcome 
(square) and the contender (circle). The standard deviation in 
this example equals once the distance between the desired 
outcome and the contender (std — 100%). 
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Figure 1: Example of data 
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The last type of parameters are those which can be con- 
trolled by the user: the minimum weight parameter /3 and the 
leaky bucket parameter £. 

Four sets of experiments were carried out. The first two 
survey the effect of each parameter using static data, the 
third examines dynamic data and the last the effect of system 
dynamics. In each experiment, all but one of the parameters 
were set to a default value: n = 10,000, \Ni\ ~ 4, a drop 
rate of zero (i.e., reliable communication), k — 3, d = 2, a 
bias of 10% and std of 100%, /3 = 0.001 and i = I. Then, 
the simulation was run ten times for each tested value of the 
remaining parameter. 

In all of the experiments in which the data is static, 
performance is measured by the number of cycles needed for 
convergence of 95% and 100% of the peers, and by the number 
of messages required for convergence of all peers (convergence 
of the last 5% of the peers usually requires very few messages). 
To allow comparison of different topologies, the average 
number of messages per edge is reported, rather than the 
grand total. When the data is dynamic, convergence never 
occurs and messages never cease. The performance metrics 
used are, therefore, the average accuracy (percent of peers 
computing the wrong outcome) and average communication 
cost (number of messages sent per communication link per 
cycle). In the literature this last metric is sometimes called 
normalized messaging and has a maximal value of two in case 
i = 1 and I in general. 

B. System properties 

For every distributed algorithm, and especially one intended 
for large systems, scalability is the single most important 
criterion. Figure |2] depicts the scale-up of LSS with selective 
uniform correction. The number of cycles required for com- 
plete quiescence and for convergence of 95% of the peers 
is depicted on the left (sub-figure |2a|i, and the number of 
messages per link is depicted on the right (sub-figure l2bl ) 

The first thing to notice is that LSS overhead seems to 
converge to a constant as the system is scaled up. This is 



certainly true for the number of cycles until 95% of the peers 
converge, and for the communication. Although convergence 
of 100% percent of the peers is an interesting metric, its 
value is mainly theoretical. First, it is a worst-case metric that 
depends on the worst performing peer Second, the typical 
working scenario of a large distributed system is dynamic, 
and does not allow 100% convergence at all. 

The second thing to notice is the instability of the perfor- 
mance when the topology is Barabasi-Albert. A deeper look 
into the results reveals that performance is greatly influenced 
by outliers: single experiments in which the overhead was 
exceptionally high. We note that Barabasi-Albert is different 
from both Chord and Grid topologies in the sense that there is 
no strict limit on peer connectivity. As can be seen in Figure 
l3bl Barabasi-Albert is also the most sensitive to average peer 
connectivity. Since each experiment is carried out using a 
constant topology, topological effects are not averaged out and 
may well explain outliers. 

Besides scale, the other important property of the system is 
connectivity, as measured in the average size of \Ni\. Because 
of the inert differences between the three topologies tested, not 
all were tested on the same range of average \Ni\. However, 
as can be seen in figures |4a] and |4b] the effect of increased 
connectivity on LSS is to expedite convergence and increase 
the number of messages per connection. Since the increase 
in communication load per link appears to be linear while 
the number of required converges quickly to a constant, there 
seems to be an optimum point, which in this experiment is 
around \Ni\ ~ 6. This could be an important observation 
because many systems do allow at least limited control of 
connectivity. 

The third and last important feature of the system is mes- 
sage reliability. In Internet based systems, reliable messaging 
usually costs very little. Even when reliability is not possible, 
message loss rate is expected to be very low. Wireless sensor 
networks are drastically different: message loss rates can be 
expected to be very high even between immediate neighbors 
and reliable messaging is usually too costly to be used for 
intensive computation. 

Figures |4a] and |4b] depict the effect of random and inde- 
pendent message loss on the convergence and the message 
overhead of LSS. As can be seen, limited message loss has 
no impact on convergence or messaging overhead. This is 
attributed to the effect of having multiple paths between peers: 
so long as corrective messages arrive through one of the paths, 
computation goes on and does converge. In all topologies, 
once a critical threshold is exceeded, convergence becomes 
impossible. This can be seen, in Figure Ha] for the Barabasi- 
Albert topology at a loss rate of more than 1 % and for the other 
topologies at a loss rate higher than 5%. In further experiments 
with a higher drop rate, Barabasi-Albert topology was always 
the most sensitive to message loss and grid topology the least 
sensitive. This supports the hypothesis on the effect multiple 
paths, because in grid topology every two neighbors are tightly 
connected through their other neighbors. 
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Figure 2: Scale-up 
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Figure 4: Message loss rate 
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C. Data properties 

Next, we investigate the sensitivity of LSS to the difficuhy 
of the problem. From previous research it is known that the 
performance of local thresholding algorithms depends mainly 
on the proximity of the average to the decision threshold. As 
can be seen in Figures |5a] and |5b] the communication overhead 
and the 95% convergence rate decrease super-exponentially 
with the bias. The 100% convergence rate also decreases with 
the bias, although perhaps not exponentially. 

Increased noise also makes computation more costly. Fig- 
ures |5c] and |5d] show that as the standard deviation is increased 
from a quarter of the distance between the desired outcome 
and the contender to four times that distance, convergence time 
grows linearly and the message overhead grows sublinearly. 

D. Ineffective parameters 

The number of possible solutions, k, and the dimensionality, 
d, of the data, can also affect performance. However, exper- 
iments with k ranging from 3 to 243 revealed no sensitivity 
of the performance. A similar result was obtained when the 
dimension of the data was varied from d ~ 1 through d ~ Q. 
We conclude that, once bias and variance are accounted for, 
neither the number of possible solutions nor the dimensionality 
of the data has any bearing on performance. We chose not to 
present these results graphicalljo 

The algorithm's parameters, the minimal weight allowed for 
\Si\, f3, and the leaky bucket period, £ were also experimented 
with. In our experiments, the algorithm underperformed when 
both were zero. However, setting the parameters to larger 
values than the default had no noticeable effect. Thus, we keep 
/? = 0.001 and £ — 1, and refrain from presenting numeric 
results. 

E. Dynamic data 

The next set of experiments is, arguably, the most realistic. 
In these experiments the data at the peers was randomly 
changed at a controlled rate for 100,000 simulation cycles. 
This hinders convergence and causes a constant need for 
further communication. Thus, the average number of peers 
which compute the wrong outcome and the average number 
of messages per link per cycle are reported instead of the 
convergence rate and total number of messages. Because 
of the large number of events in these experiments, they 
were executed on networks of just 1000 peers. Also, these 
experiments were carried out with twice the default bias (20% 
rather than 10%) and twice the default standard deviation 
of the data (twice the distance of the desired outcome from 
the contender rather than just once that distance) in order to 
increase the effect of every change. 

As can be seen in Figures |6a] and |6b] up to a noise 
rate in which one peer's input changes, on average, at each 
simulator cycle, the effect of data dynamics is almost only on 
communication and not on accuracy. Correction is, apparently, 
fast enough so that the occasional changed input does not 

^As said earlier, code will be provided for anyone who wishes to verify 
these results. 



propagate an error to a larger part of the network. On the 
other hand, that same correction does cost in messages. So 
communication cost does grow about linearly with the noise 
rate. Then, at about the noise rate which provides for one 
change in each simulator cycle, the effect of the change 
on accuracy begins to become noticeable, and the errors 
accumulate linearly with the noise rate. 

The effect of message loss in a dynamic setup is different 
than it is when the data is static. In a static setup, a peer who 
does not receive the intended message does not react to correct 
the wrong output of the sender. In a dynamic setup, however, 
a peer has many more triggers that will cause it to react, and 
message loss has only a short-term effect on correctness. 

Evidence for this can be seen in Figure |7a] and Figure |7b] In 
these experiments, the loss rate is gradually increased, and the 
data is dynamically changed at a rate of one thousand peers 
per million per simulator cycle. As can be seen, the percentage 
of peers which compute the wrong result is extremely low. 
This means that the errors induced by message loss hardly 
accumulate. When 5% of all messages are lost, the error rate 
is less than half percent. In comparison, in Figure I4a1 one can 
see that at this loss rate the error rate at an experiment with 
static data skyrockets. 

The second phenomenon evident in this dynamic setup 
is that the variance of both the accuracy (Figure |7a) and 
the communication overhead (Figure ITbT i is very large in 
Barabasi-Albert and in Chord topologies, but not in the Grid 
topology. Again, our hypothesis is that a greater number of 
short alternate routes between every two neighbors increases 
the algorithm's robustness to message loss. 

F. Dynamic network 

Finally, the robustness of the LSS algorithm to peer churn 
is validated. Again, a network of 2,000 peers is simulated 
for 100,000 cycles with their data changing at a rate of 
1,000 ppmc. Additionally, peers drop out of the system at 
a controlled rate of between zero (no churn) and four ppmc. 
It is assumed that churn is detected by the peer's neighbors, 
which then recalculate their status and correct as needed. 

Figures|8a]and[8b]depict the effect different chum rates have 
on the average error and the average message load per link. 
Beside the chum rate, the x-axis also denotes the percentage 
of peers remaining active at the end of the 100,000 cycles. 
As can be seen in Figure (Ha] the error rate grows notably as 
more peers chum. However, even when eventual churn nears 
40% of the peers, no more than 1% of them compute the 
wrong outcome on average. Message overhead increases with 
churn, probably due to the increased effort needed to correct 
the mistaken outcome. However, the trend is not very clear and 
the overhead is very noisy. It is worth noting that the regularity 
of the bi-dimensional grid is degraded with churn, which may 
explain why performance is similar in that topology and in the 
other two topologies even though they are less regular. 

VII. Related work 

This paper makes a fundamental contribution to computa- 
tion in large distributed systems. As such, it relates to other 



10 



■ Barabasi-Albert - 95% 
3arabasi-Albert - 100% 
Chord - 95% 

3hord - 1 00% 

■ 3rid - 95%. 

end -100% 





(a) Sensitivity to bias- convergence 



(b) Sensitivity to bias - communication 



: IBarabasi-Albert - 95%< 
■ Barabasi-Albert - 100%. 

3hord - 95% 

Ghord - 1 00% 

3rid - 95% 

3nd-100% 




1% (S^. 

Standard deviations between desired outcome and contender 



(c) Sensitivity to variance - convergence 




standard deviations between desired outcome and contender 
(d) Sensitivity to variance - communication 



Figure 5: Problem difficulty 




Data changes per peer per million simulation cycle 

(a) Average percent of errors 



Data changes per peer per million simulation cycle 

(b) Average number of messages per cycle per connection 



Figure 6: Dynamically changing data 
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Figure 7: Message loss and dynamically changing data 




methods of computation in similar networks. We choose to 
categorize such methods according to the regime used to 
dictate which messages are sent. 

The first category is algorithms that enforce a strict mes- 
saging regime. This category includes convergecast-based in- 
network computation (e.g., those using MapReduce (2T\), 
which have been used for years in small-scale distributed 
systems. It also includes algorithms such as |fT9l . in which 
messages flow through the entire system in a strict order. 
As systems grow larger, global methods lose their appeal. 
This is mainly because enforcing order, synchronization, and 
reliability becomes impermissibly costly. 

The second category is algorithms that are based on re- 
peated averaging. General results about the convergence of 
statistics under repeated averaging are known since the 70' s 
|l9l. They were first implemented for function computation in 
a distributed system in applications such as diffusive load 
balancing IT31 . averaging [[33], and Kalman filtering ll26l . 



The first relation of repeated averaging to distributed data 
mining in peer-to-peer system was apparently in the context 
of the DREAM project fVl]. Diffusion has also been shown to 
allow solving more general optimization problems than merely 
averaging ll25l. ifTOl. 

Kempe et al., however, were the first to position repeated 
averaging in the context of gossip algorithms and to provide 
much needed bounds on convergence speed llT6l . 0, lfT4l . 
El, IIT2I . Gossip based algorithms were shown to converge 
with the logarithm of the network size if uniformly random 
messaging is possible. Otherwise, their convergence rate de- 
pends on the eigen-gap of the network graph ||6]. Gossip- 
based algorithms are also simple and robust and have been 
applied to a large number of problems (see, e.g., ifTTI . ||221 ). 
However, since, at base, their convergence rate depends on the 
random mixing of inputs, gossip algorithms are still extremely 
wasteful. In a wireless sensor network, where the messaging 
budget is scarce, it seems inherently wrong to send messages 
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at random. 

Local thresholding defines the last category of algorithms. 
Unlike gossip-based algorithms, local algorithms are deter- 
ministic. They are also far more data dependent. There are 
ample situations in which a peer running local thresholding 
will not send any message at all. Comparative testing of local 
thresholding and gossip algorithms ||32l shows that the former 
are vastly more efficient. 

Previous local thresholding algorithms all rely on the cycle 
freedom of the network. The proof of the stopping condition 
they use relies on the fact that the latest message received 
from a neighbor pj G Ni does not depend on the input of 
Pi or on inputs which are accounted for in messages received 
from other pk E Ni. 

Besides the difficulty of providing a cycle free network, this 
also meant those algorithms were critically dependent on the 
reliability of messages to neighbors. If a message is lost on the 
way from pi to pj then there is no alternative path in which 
the inputs represented in that message can propagate the pj. 
This dependency has made them even less suitable to wireless 
sensor network applications. This work is the first to lift the 
requirement of cycle freedom. 

VIII. Conclusion and further research 

The new local stopping rule and update methods presented 
in this paper remove a difficult barrier to the implementation 
of local thresholding algorithms in some of the most popular 
distributed environments. It is further hoped that the novel 
presentation of these algorithms as operating in the field 
of weighted vectors will simplify future developments. As 
demonstrated here, local thresholding can be an extremely effi- 
cient way to compute complex functions over large distributed 
networks, regardless of their topological characteristics. 

Recent years have seen a lot of focus on applications of 
local thresholding algorithms and on their stopping rules. 
This work, and especially the presentation of the problem in 
terms of weighted vectors, greatly simplifies the argument in 
favor of certain update policies. We believe some interesting 
problems lie in the correction poUcy. Although much important 
work has been done on expediting convergence of gossip 
algorithms and diffusive load balancing |13J algorithms using 
smart update rules, we are not aware of any parallel work on 
local thresholding algorithms. 

The generalization described here still misses one interest- 
ing aspect of some of the target systems: that the commu- 
nication graph is often directed and weighted. In a wireless 
sensor network, the signal transmitted by a sensor may well 
be received by a sensor whose signal it cannot receive. The 
might also have different energy levels, meaning messages are 
more costly to one than they are to the other In structured 
peer-to-peer systems such as Chord, routing from a peer to a 
peer in its finger table costs just one message whereas routing 
in the opposite direction can cost log (rt) messages. In general, 
expediting convergence by taking into account message delays, 
and not merely connectivity, may be an important challenge. 
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